Image sensor, image-capturing apparatus and image-capturing system

ABSTRACT

To produce both 2D image data and color parallax image data from image data output from a single-plate image sensor, the resolutions of the 2D image data and parallax image data may be both degraded. The image sensor has a primitive lattice that is a group of pixels including (i) at least four types of parallax pixels formed by photoelectric converter elements each of which is associated with one of combinations of two different types of aperture masks and two different types of color filters and (ii) no-parallax pixels configured to guide an incident luminous flux to photoelectric converter elements without limitation. In the group of pixels, the no-parallax pixels are more than the parallax pixels.

This application is a divisional application of U.S. patent applicationSer. No. 14/487,554 filed on Sep. 16, 2014, which in turn claimspriority to the following Japanese patent applications:

No. 2012-060737 filed on Mar. 16, 2012,

No. 2012-060738 filed on Mar. 16, 2012,

No. 2012-182417 filed on Aug. 21, 2012,

No. 2012-182420 filed on Aug. 21, 2012, and

PCT/JP2013/001811 filed on Mar. 15, 2013.

The contents of all applications are incorporated herein by reference intheir entirety.

BACKGROUND 1. Technical Field

The present invention relates to an image sensor, an image-capturingapparatus and an image-capturing system.

2. Related Art

A known stereo image-capturing apparatus uses two image-capturingoptical systems to capture a stereo image consisting of a left-eye imageand a right-eye image. Such a stereo image-capturing apparatus has thetwo image-capturing optical systems arranged with a predetermineddistance provided therebetween so as to generate parallax between thetwo images obtained by imaging the same subject.

Prior Art Document

Patent Document 1: Japanese Patent Application Publication No. 8-47001

When image data output from a single-chip image sensor is used tosimultaneously produce 2D image data and color parallax image data, theresolutions of the 2D image data and the parallax image data may be bothadversely affected.

SUMMARY

A first aspect of the innovations may include an image sensor having aprimitive lattice formed by a group of pixels including at least fourtypes of parallax pixels having photoelectric converter elements each ofwhich is associated with (i) one of a first aperture mask and a secondaperture mask that respectively have openings positioned to transmitdifferent partial luminous fluxes of an incident luminous flux from eachother and (ii) one of a first color filter and a second color filterthat respectively transmit different wavelength ranges from each other,and no-parallax pixels configured to guide the incident luminous flux tophotoelectric converter elements without limitation by openings. Here,in the group of pixels, the no-parallax pixels are more than theparallax pixels.

A second aspect of the innovations may include an image-capturingapparatus including the above-described image sensor, and an imageprocessor configured to produce a plurality of pieces of parallax imagedata having parallax therebetween and 2D image data without parallaxbased on the outputs from the image sensor.

A third aspect of the innovations may include an image sensor includingparallax pixels having photoelectric converter elements each of which isassociated with one of a plurality of types of aperture masks thatrespectively have openings positioned to transmit different partialluminous fluxes of an incident luminous flux from each other, andno-parallax pixels configured to guide the incident luminous flux tophotoelectric converter elements without limitation of openings. Here,the parallax pixels are arranged at equal intervals in both of twodimensional directions in such a manner that a parallax pixel associatedwith an aperture mask of one of the types is sandwiched between ano-parallax pixel and a parallax pixel associated with an aperture maskof a different one of the types, and parallax pixels associated withaperture masks of different ones of the types are arranged as distant aspossible.

A fourth aspect of the innovations may include an image-capturingapparatus including the above-described image sensor, and an imageprocessor configured to produce a plurality of pieces of parallax imagedata having parallax therebetween and 2D image data without parallaxbased on the outputs from the image sensor.

The summary clause does not necessarily describe all necessary featuresof the embodiments of the present invention. The present invention mayalso be a sub-combination of the features described above. The above andother features and advantages of the present invention will become moreapparent from the following description of the embodiments taken inconjunction with the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates the structure of a digital camera relating to anembodiment of the present invention.

FIG. 2 is a cross-sectional view illustrating the structure of an imagesensor relating to an embodiment of the present invention.

FIG. 3 schematically illustrates an enlarged portion of the imagesensor.

FIGS. 4A, 4B and 4C are conceptual views to illustrate the relationbetween a parallax pixel and a subject.

FIG. 5 is a conceptual view to illustrate how to produce a parallaximage.

FIG. 6 illustrates a Bayer arrangement.

FIG. 7 illustrates how pixels are arranged in a repeating pattern 110 ina first implementation.

FIG. 8 illustrates how the pixel pitches of the various types ofparallax pixels are related to each other in the first implementation.

FIG. 9 illustrates how the pixels are arranged in a repeating pattern110 relating to a second implementation.

FIG. 10 illustrates how the pixels are arranged in a repeating pattern110 relating to a third implementation.

FIG. 11 illustrates how the pixels are arranged in a repeating pattern110 relating to a fourth implementation.

FIG. 12 illustrates, as an example, how to produce RGB plane data, whichis 2D image data.

FIG. 13 illustrates, as an example, how to produce two pieces of G planedata, which are parallax image data.

FIG. 14 illustrates, as an example, how to produce two pieces of B planedata, which are parallax image data.

FIG. 15 illustrates, as an example, how to produce two pieces of R planedata, which are parallax image data.

FIG. 16 is a conceptual view to illustrate the relation between theresolutions of the respective planes.

FIGS. 17A and 17B illustrate the concept of defocusing.

FIGS. 18A and 18B illustrate the concept of defocusing for parallaxpixels.

FIGS. 19A and 19B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 20A and 20B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 21A and 21B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 22A and 22B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 23A and 23B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 24A and 24B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 25A and 25B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 26A and 26B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 27A and 27B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 28A and 28B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 29A and 29B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 30A and 30B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 31A and 31B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 32A and 32B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIG. 33 illustrates a full-open no-parallax pixel and a half-openno-parallax pixel.

FIGS. 34A and 34B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 35A and 35B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIGS. 36A and 36B illustrate, as an example, an arrangement in the realspace and a corresponding k-space.

FIG. 37 shows moving image reading.

DESCRIPTION OF EXEMPLARY EMBODIMENTS

Hereinafter, some embodiments of the present invention will bedescribed. The embodiments do not limit the invention according to theclaims, and all the combinations of the features described in theembodiments are not necessarily essential to means provided by aspectsof the invention.

A digital camera relating to the present embodiment, which is a form ofan image processing apparatus and an image-capturing apparatus, isconfigured to be capable of producing for a single scene a plurality ofimages from a plurality of viewpoints with a single image-capturingoperation. Here, the images from different viewpoints are referred to asparallax images.

FIG. 1 illustrates the structure of a digital camera 10. The digitalcamera 10 includes an image-capturing lens 20, which is animage-capturing optical system, and guides incoming subject luminousflux along an optical axis 21 to an image sensor 100. Theimage-capturing lens 20 may be a replaceable lens that is attachable anddetachable to/from the digital camera 10. The digital camera 10 includesthe image sensor 100, a controller 201, an A/D converter circuit 202, amemory 203, a drive unit 204, an image processor 205, a memory card IF207, an operating unit 208, a display 209, an LCD driver circuit 210,and an AF sensor 211.

As shown in FIG. 1, a z-axis positive direction is defined as thedirection parallel to the optical axis 21 toward the image sensor 100,an x-axis positive direction is defined as the direction toward theviewer of the sheet of FIG. 1 in the plane orthogonal to the z axis, anda y-axis positive direction is defined as the upward direction in thesheet of FIG. 1. In some of the following drawings, their coordinateaxes are shown how the respective drawings are arranged relative to thecoordinate axes of FIG. 1.

The image-capturing lens 20 is constituted by a group of optical lensesand configured to form an image from the subject luminous flux from ascene in the vicinity of its focal plane. For the convenience ofdescription, the image-capturing lens 20 is hypothetically representedby a single lens positioned in the vicinity of the pupil in FIG. 1. Theimage sensor 100 is positioned in the vicinity of the focal plane of theimage-capturing lens 20. The image sensor 100 is an image sensor havinga two-dimensionally arranged photoelectric converter elements, forexample, a CCD or CMOS sensor. The timing of the image sensor 100 iscontrolled by the driver unit 204 so that the image sensor 100 canconvert a subject image formed on the light receiving surface into animage signal and outputs the image signal to the A/D converter circuit202.

The A/D converter circuit 202 converts the image signal output from theimage sensor 100 into a digital image signal and outputs the digitalimage signal to the memory 203. The image processor 205 uses the memory203 as its workspace to perform a various image processing operationsand thus generates image data.

The image processor 205 additionally performs general image processingoperations such as adjusting image data in accordance with a selectedimage format. The produced image data is converted by the LCD drivecircuit 210 into a display signal and displayed on the display 209. Inaddition, the produced image data is stored in the memory card 220attached to the memory card IF 207.

The AF sensor 211 is a phase detection sensor having a plurality ofranging points set in a subject space and configured to detect a defocusamount of a subject image for each ranging point. A series ofimage-capturing sequences is initiated when the operating unit 208receives a user operation and outputs an operating signal to thecontroller 201. The various operations such as AF and AE associated withthe image-capturing sequences are performed under the control of thecontroller 201. For example, the controller 201 analyzes the detectionsignal from the AF sensor 211 to perform focus control to move a focuslens that constitutes a part of the image-capturing lens 20.

The following describes the structure of the image sensor 100 in detail.FIG. 2 schematically illustrates the cross-section of the image sensor100 relating to an embodiment of the present invention.

The image sensor 100 is structured in such a manner that microlenses101, color filters 102, aperture masks 103, an interconnection layer 105and photoelectric converter elements 108 are arranged in the statedorder when seen from the side facing a subject. The photoelectricconverter elements 108 are formed by photodiodes that may convertincoming light into an electrical signal. The photoelectric converterelements 108 are arranged two-dimensionally on the surface of asubstrate 109.

The image signals produced by the conversion performed by thephotoelectric converter elements 108, control signals to control thephotoelectric converter elements 108 and the like are transmitted andreceived via interconnections 106 provided in the interconnection layer105. The aperture masks 103 having openings 104, which are provided in aone-to-one correspondence with the photoelectric converter elements 108,are provided in contact with the interconnection layer 105. Each of theopenings 104 is shifted in accordance with a corresponding one of thephotoelectric converter elements 108 and strictly positioned relative tothe corresponding photoelectric converter element 108 as describedlater. As described later in more details, the aperture masks 103 havingthe openings 104 effectively cause parallax in the subject luminous fluxreceived by the photoelectric converter elements 108.

On the other hand, no aperture masks 103 are provided on some of thephotoelectric converter elements 108 that do not cause parallax. Inother words, such photoelectric converter elements 108 are provided withthe aperture masks 103 having such openings 104 that do not limit thesubject luminous flux incident on the corresponding photoelectricconverter elements 108 or allow the entire incident luminous flux totransmit through the aperture masks 103. Although these photoelectricconverter elements 108 do not cause parallax, the incoming subjectluminous flux is substantially defined by an opening 107 formed by theinterconnections 106. Therefore, the interconnections 106 can be viewedas an aperture mask that does not cause parallax and allows the entireincoming luminous flux to pass. The aperture masks 103 may be arrangedindependently and separately from the photoelectric converter elements108 and in correspondence with the photoelectric converter elements 108,or may be formed jointly with the photoelectric converter elements 108,like the way how the color filters 102 are manufactured.

The color filters 102 are provided on the aperture masks 103. Each ofthe color filters 102 is colored so as to transmit a particularwavelength range to a corresponding one of the photoelectric converterelements 108, and the color filters 102 are arranged in a one-to-onecorrespondence with the photoelectric converter elements 108. To outputa color image, at least two different types of color filters that aredifferent from each other need to be arranged. However, three or moredifferent types of color filters may need to be arranged to produce acolor image with higher quality. For example, red filters (R filters) totransmit the red wavelength range, green filters (G filters) to transmitthe green wavelength range, and blue filters (B filters) to transmit theblue wavelength range may be arranged in a lattice pattern. The way howthe filters are specifically arranged will be described later.

The microlenses 101 are provided on the color filters 102. Themicrolenses 101 are each a light collecting lens to guide more of theincident subject luminous flux to the corresponding photoelectricconverter element 108. The microlenses 101 are provided in a one-to-onecorrespondence with the photoelectric converter elements 108. Theoptical axis of each microlens 101 is preferably shifted so that more ofthe subject luminous flux is guided to the corresponding photoelectricconverter element 108 taking into consideration the relative positionsbetween the pupil center of the image-capturing lens 20 and thecorresponding photoelectric converter element 108. Furthermore, theposition of each of the microlenses 101 as well as the position of theopening 104 of the corresponding aperture mask 103 may be adjusted toallow more of the particular subject luminous flux to be incident, whichwill be described later.

Here, a pixel is defined as a single set constituted by one of theaperture masks 103, one of the color filters 102, and one of themicrolenses 101, which are provided in a one-to-one correspondence withthe photoelectric converter elements 108 as described above. To be morespecific, a pixel with an aperture mask 103 that causes parallax isreferred to as a parallax pixel, and a pixel without an aperture mask103 that causes parallax is referred to as a no-parallax pixel. Forexample, when the image sensor 100 has an effective pixel region ofapproximately 24 mm×16 mm, the number of pixels reaches as many asapproximately 12 million.

When image sensors have high light collection efficiency andphotoelectric conversion efficiency, the microlenses 101 may be omitted.Furthermore, in the case of back side illumination image sensors, theinterconnection layer 105 is provided on the opposite side of thephotoelectric converter elements 108. In addition, the color filters 102and the aperture masks 103 can be integrally formed by allowing theopenings 104 of the aperture masks 103 to have color components.

In the present embodiment, the aperture masks 103 are separately formedfrom the interconnections 106, but the function of the aperture masks103 in the parallax pixels may be alternatively performed by theinterconnections 106. In other words, defined opening shapes are formedby the interconnections 106 and limit the incident luminous flux toallow only particular partial luminous flux to pass to reach thephotoelectric converter elements 108. In this case, the interconnections106 forming the opening shapes are preferably positioned closest to thephotoelectric converter elements 108 in the interconnection layer 105.

The aperture masks 103 may be formed by a transmission preventing filmthat is overlaid on the photoelectric converter elements 108. In thiscase, the aperture masks 103 are formed in such a manner that, forexample, a SiN film and a SiO₂ film are sequentially stacked to form atransmission preventing film and regions corresponding to the openings104 are removed by etching.

The following describes the relation between the openings 104 of theaperture masks 103 and parallax caused. FIG. 3 schematically illustratesan enlarged portion of the image sensor 100. To simplify thedescription, the arrangement of the colors of the color filters 102 isnot considered here, which will be mentioned later. In the followingdescription where the arrangement of the colors of the color filters 102is ignored, the image sensor 100 may be viewed as an image sensor thatis constituted only by parallax pixels having the color filters 102 ofthe same color. Accordingly, the repeating pattern described in thefollowing may be viewed as applied to the adjacent pixels among thecolor filters 102 of the same color.

As shown in FIG. 3, the openings 104 of the aperture masks 103 areshifted relative to the corresponding pixels. Furthermore, the openings104 of the adjacent pixels are positioned differently from each other.

In the example shown in FIG. 3, six different types of aperture masks103 are provided, in which the openings 104 are shifted in the left andright directions relative to the corresponding pixels. When the imagesensor 100 as a whole is considered, photoelectric converter elementgroups, each of which has six parallax pixels having the aperture masks103, are two-dimensionally and periodically arranged such that theopenings 104 are shifted from left to right on the sheet of FIG. 3.

FIGS. 4A, 4B and 4C are each a conceptual diagram illustrating therelation between parallax pixels and a subject. To be specific, FIG. 4Aillustrates a photoelectric converter element group having a repeatingpattern 110 t arranged at the center of the image sensor 100 throughwhich the image-capturing optical axis 21 extends. FIG. 4B schematicallyillustrates a photoelectric converter element group having a repeatingpattern 110 u of the parallax pixels arranged in the peripheral portionof the image sensor 100. In FIGS. 4A and 4B, a subject 30 is positionedat a focus position relative to the image-capturing lens 20. FIG. 4Cschematically illustrates the relation between the parallax pixels andthe subject when a subject 31 at a non-focus position relative to theimage-capturing lens 20 is captured, correspondingly to the relationshown in FIG. 4A.

The following first describes the relation between the parallax pixelsand the subject when the image-capturing lens 20 captures the subject 30at the focus position. The subject luminous flux is guided through thepupil of the image-capturing lens 20 to the image sensor 100. Here, sixpartial regions Pa to Pf are defined in the entire cross-sectionalregion through which the subject luminous flux transmits. For example,see the pixel, on the extreme left in the sheet of FIGS. 4A to 4C, ofthe photoelectric converter element groups having the repeating patterns110 t and 110 u. The position of the opening 104 f of the aperture mask103 is defined so that only the subject luminous flux emitted from thepartial region Pf reaches the photoelectric converter element 108 asseen from the enlarged view. Likewise, towards the pixel on the farright, the position of the opening 104 e is defined so as to correspondto the partial region Pe, the position of the opening 104 d is definedso as to correspond to the partial region Pd, the position of theopenings 104 c is defined so as to correspond to the partial region Pc,the position of the openings 104 b is defined so as to correspond to thepartial region Pb, and the position of the opening 104 a is defined soas to correspond to the partial region Pa.

Stated differently, for example, the gradient of the main light ray Rfof the subject luminous flux (partial luminous flux) emitted from thepartial region Pf, which is defined by the relative positions of thepartial region Pf and the leftmost pixel, may determine the position ofthe opening 104 f. When the photoelectric converter element 108 receivesthe subject luminous flux through the opening 104 f from the subject 30at the focus position, the subject luminous flux forms an image on thephotoelectric converter element 108 as indicated by the dotted line.Likewise, toward the rightmost pixel, the gradient of the main light rayRe determines the position of the opening 104 e, the gradient of themain light ray Rd determines the position of the opening 104 d, thegradient of the main light ray Rc determines the position of the opening104 c, the gradient of the main light ray Rb determines the position ofthe opening 104 b, and the gradient of the main light ray Ra determinesthe position of the opening 104 a.

As shown in FIG. 4A, the luminous flux emitted from a micro region Ot ofthe subject 30 at the focus position, which coincides with the opticalaxis 21 on the subject 30, passes through the pupil of theimage-capturing lens 20 and reaches the respective pixels of thephotoelectric converter element group having the repeating pattern 110t. In other words, the pixels of the photoelectric converter elementgroup having the repeating pattern 110 t respectively receive theluminous flux emitted from the single micro region Ot through the sixpartial regions Pa to Pf. The micro region Ot has a certain spread thatcan accommodate the different positions of the respective pixels of thephotoelectric converter element group having the repeating pattern 110t, but can be substantially approximated by substantially the sameobject point. Likewise, as shown in FIG. 4B, the luminous flux emittedfrom a micro region Ou of the subject 30 at the focus position, which isspaced away from the optical axis 21 on the subject 30, passes throughthe pupil of the image-capturing lens 20 to reach the respective pixelsof the photoelectric converter element group having the repeatingpattern 110 u. In other words, the respective pixels of thephotoelectric converter element group having the repeating pattern 110 urespectively receive the luminous flux emitted from the single microregion Ou through the six partial regions Pa to Pf. Like the microregion Ot, the micro region Ou has a certain spread that can accommodatethe different positions of the respective pixels of the photoelectricconverter element group having the repeating pattern 110 u, but can besubstantially approximated by substantially the same object point.

That is to say, as long as the subject 30 is at the focus position, thephotoelectric converter element groups capture different micro regionsdepending on the positions of the repeating patterns 110 on the imagesensor 100, and the respective pixels of each photoelectric converterelement group capture the same micro region through the differentpartial regions. In the respective repeating patterns 110, thecorresponding pixels receive subject luminous flux from the same partialregion. To be specific, in FIGS. 4A and 4B, for example, the leftmostpixels of the repeating patterns 110 t and 110 u receive the partialluminous flux from the same partial region Pf.

Strictly speaking, the position of the opening 104 f of the leftmostpixel that receives the subject luminous flux from the partial region Pfin the repeating pattern 110 t at the center through which theimage-capturing optical axis 21 extends is different from the positionof the opening 104 f of the leftmost pixel that receives the subjectluminous flux from the partial region Pf in the repeating pattern 110 uat the peripheral portion. From the perspective of the functions,however, these openings can be treated as the same type of aperturemasks in that they are both aperture masks to receive the subjectluminous flux from the partial region Pf. Accordingly, in the exampleshown in FIGS. 4A to 4C, it can be said that each of the parallax pixelsarranged on the image sensor 100 has one of the six types of aperturemasks.

The following describes the relation between the parallax pixels and thesubject when the image-capturing lens 20 captures the subject 31 at thenon-focus position. In this case, the subject luminous flux from thesubject 31 at the non-focus position also passes through the six partialregions Pa to Pf of the pupil of the image-capturing lens 20 to reachthe image sensor 100. However, the subject luminous flux from thesubject 31 at the non-focus position forms an image not on thephotoelectric converter elements 108 but at a different position. Forexample, as shown in FIG. 4C, when the subject 31 is at a more distantposition from the image sensor 100 than the subject 30 is, the subjectluminous flux forms an image at a position closer to the subject 31 withrespect to the photoelectric converter elements 108. On the other hand,when the subject 31 is at a position closer to the image sensor 100 thanthe subject 30 is, the subject luminous flux forms an image at aposition on the opposite side of the subject 31 with respect to thephotoelectric converter elements 108.

Accordingly, the subject luminous flux emitted from a micro region Ot′of the subject 31 at the non-focus position reaches the correspondingpixels of different repeating patterns 110 depending on which of the sixpartial regions Pa to Pf the subject luminous flux passes through. Forexample, the subject luminous flux that has passed through the partialregion Pd enters the photoelectric converter element 108 having theopening 104 d included in the repeating pattern 110 t′ as a main lightray Rd′ as shown in the enlarged view of FIG. 4C. The subject luminousflux that has passed through the other partial regions may be emittedfrom the micro region Ot′, but does not enter the photoelectricconverter elements 108 included in the repeating pattern 110 t′ andenters the photoelectric converter elements 108 having the correspondingopenings in different repeating patterns. In other words, the subjectluminous fluxes that reach the respective photoelectric converterelements 108 constituting the repeating pattern 110 t′ are subjectluminous fluxes emitted from different micro regions of the subject 31.To be specific, the subject luminous flux having the main light ray Rd′enters the photoelectric converter element 108 corresponding to theopening 104 d, and the subject luminous fluxes having the main lightrays Ra⁺, Rb⁺, Rc⁺, Re⁺, Rf⁺, which are emitted from different microregions of the subject 31, enter the photoelectric converter elements108 corresponding to the other openings 104. The same relation is alsoseen in the repeating pattern 110 u arranged in the peripheral portionshown in FIG. 4B.

Here, when the image sensor 100 is seen as a whole, for example, asubject image A captured by the photoelectric converter element 108corresponding to the opening 104 a and a subject image D captured by thephotoelectric converter element 108 corresponding to the opening 104 dmatch with each other if they are images of the subject at the focusposition, and do not match with each other if they are images of thesubject at the non-focus position. The direction and amount of thenon-match are determined by on which side the subject at the non-focusposition is positioned with respect to the focus position, how much thesubject at the non-focus position is shifted from the focus position,and the distance between the partial region Pa and the partial regionPd. Stated differently, the subject images A and D are parallax imagescausing parallax therebetween. This relation also applies to the otheropenings, and six parallax images are formed corresponding to theopenings 104 a to 104 f.

Accordingly, a collection of outputs from the corresponding pixels indifferent ones of the repeating patterns 110 configured as describedabove produces a parallax image. To be more specific, the outputs fromthe pixels that have received the subject luminous flux emitted from aparticular partial region of the six partial regions Pa to Pf form aparallax image.

FIG. 5 is a conceptual diagram to illustrate an operation to produce aparallax image. FIG. 5 shows, from left to right, how parallax imagedata Im_f is produced by collecting the outputs from the parallax pixelscorresponding to the openings 104 f, how parallax image data Im_e isproduced from the outputs of the parallax pixels corresponding to theopenings 104 e, how parallax image data Im_d is produced from theoutputs of the parallax pixels corresponding to the openings 104 d, howparallax image data Im_c is produced from the outputs of the parallaxpixels corresponding to the openings 104 c, how parallax image data Im_bis produced from the outputs of the parallax pixels corresponding to theopenings 104 b, and how parallax pixel data Im_a is produced from theoutputs from the parallax pixels corresponding to the openings 104 a.The following first describes how parallax image data Im_f is producedby collecting the outputs from the parallax pixels corresponding to theopenings 104 f.

The repeating patterns 110 each of which has a photoelectric converterelement group constituted by a group of six parallax pixels are arrangedside-by-side. Accordingly, on the hypothetical image sensor 100excluding no-parallax pixels, the parallax pixels having the openings104 f are found every six pixels in the horizontal direction andconsecutively arranged in the vertical direction. These pixels receivesubject luminous fluxes from different micro regions as described above.Therefore, parallax images can be obtained by collecting and arrangingthe outputs from theses parallax pixels.

However, the pixels of the image sensor 100 of the present embodimentare square pixels. Therefore, if the outputs are simply collected, thenumber of pixels in the horizontal direction is reduced to one-sixth andvertically long image data is produced. To address this issue,interpolation is performed to increase the number of pixels in thehorizontal direction six times. In this manner, the parallax image dataIm_f is produced as an image having the original aspect ratio. Notethat, however, the horizontal resolution is lower than the verticalresolution since the parallax image data before the interpolationrepresents an image whose number of pixels in the horizontal directionis reduced to one-sixth. In other words, the number of pieces ofparallax image data produced is inversely related to the improvement ofthe resolution. The interpolation applied in the present embodiment willbe specifically described later.

In the similar manner, parallax image data Im_e to parallax image dataIm_a are obtained. Stated differently, the digital camera 10 can produceparallax images from six different viewpoints with horizontal parallax.

The above has described an exemplary case where the openings of theparallax pixels have six different types of position shifts with respectto the center of the pixels and the different types of parallax pixelsfunction as a pupil-division optical system to provide for parallaxpixels of six viewpoints. The above mainly describes that the respectivetypes of parallax pixels form subject images of different viewpoints andthe different types of parallax pixels cause parallax between imagestaken from different viewpoints.

A pupil-division optical system using parallax pixels not only causesparallax but also, more importantly, causes parallax only between blurimages of a subject at a non-focus position that is off the focusposition according to the degree of the non-focus.

This fact is described for a normal no-parallax pixel and two pixels(left and right pixels) with reference to FIGS. 17A and 17B and FIGS.18A and 18B. Light, which has passed through a single optical system,forms an optical image as if the hypothetical pupil is provided on theleft side of the optical system in the case of a right parallax pixeland forms an optical image as if the hypothetical pupil is provided onthe right-hand side of the optical system in the case of a left parallaxpixel. Accordingly, the point spread function of the image of a subjectat the focus position is sharp, and the light from the subject at thefocus position forms a no-parallax subject image having a sharp pointspread irrespective of whether the light passes through either of thehypothetical pupils. On the other hand, the point spread function of asubject at a position that is off the focus position toward or away fromthe optical system is broad. Furthermore, as the subject is moved awayfrom the focus position, the blur width of the subject image increasesand the centers of the respective formed images are further spaced awayfrom each other in the left-and-right direction, which causes parallax.When these two separate point spread functions are added and combinedtogether, the resulting single point spread function matches with thepoint spread function of the image formed by the no-parallax pixel, andthe peak of the resulting single point spread function is positionedbetween and equally distant from the point spread functions of the twoseparate optical images formed by the hypothetical pupils.

The present embodiment takes advantage of the important fact thatparallax only exists in the blur and the disparity increases as the blurincreases. The present embodiment proposes a color and parallaxarrangement that is designed to simultaneously obtain high-definition 2Dand 3D images.

The following describes the color filters 102 and the parallax images.FIG. 6 illustrates a Bayer arrangement. As shown in FIG. 6, G filtersare assigned to the two pixels, i.e., the upper-left (Gb) and lowerright (Gr) pixels, an R filter is assigned to one pixel, i.e., the lowerleft pixel, and a B filter is assigned to one pixel, i.e., an upperright pixel in the Bayer arrangement.

Based on such an arrangement of the color filters 102, an enormousnumber of different repeating patterns 110 can be defined depending onto what colors the parallax and no-parallax pixels are allocated and howfrequently parallax and no-parallax pixels are allocated. Collecting theoutputs of the no-parallax pixels can produce no-parallax captured imagedata like an ordinary captured image. Accordingly, a high-resolution 2Dimage can be output by increasing the ratio of the no-parallax pixelsrelative to the parallax pixels. In this case, the ratio of the parallaxpixels decreases relative to the no-parallax pixels and a 3D imageformed by a plurality of parallax images exhibits lower image quality.On the other hand, if the ratio of the parallax pixels increases, the 3Dimage exhibits improved image quality. However, since the ratio of theno-parallax pixels decreases relative to the parallax pixels, alow-resolution 2D image is output. If the parallax pixels are allocatedto all of the R, G and B pixels, the resulting color image datarepresents a 3D image having excellent color reproducibility and highquality.

Irrespective of whether the color image data represents a 2D or 3Dimage, the color image data ideally has high resolution and quality.Here, the region of a 3D image for which an observer senses parallaxwhen observing the 3D image is the non-focus region in which theidentical subject images do not match, as understood from the cause ofthe parallax, which is described with reference to FIGS. 4A to 4C, 17Aand 17B, and 18A and 18B. This means that, in the region of the image inwhich the observer senses parallax, fewer high-frequency components arepresent than in the focused image of the main subject. Considering this,the image data required to produce a 3D image does not need to have veryhigh resolution in the region in which parallax is generated.

Regarding the focused region of the image, the corresponding image datais extracted from 2D image data. Regarding the non-focused region of theimage, the corresponding image data is extracted from 3D image data. Inthis way, parallax image data can be produced by combining these piecesof image data for the focused and non-focused regions. Alternatively,high-resolution 2D image data is used as basic data and multiplied bythe relative ratios of the 3D image data on the pixel-by-pixel basis. Inthis way, high-resolution parallax image data can be produced. When suchimage processing is employed, the number of the parallax pixels may beallowed to be smaller than the number of the no-parallax pixels in theimage sensor 100. In other words, a 3D image having a relatively highresolution can be produced even if the number of the parallax pixels isrelatively small.

In this case, to produce the 3D image in color, at least two differenttypes of color filters may need to be arranged. In the presentembodiment, however, three types of, i.e., R, G and B color filters areemployed as in the Bayer arrangement described with reference to FIG. 6in order to further improve the image quality. To be specific, in thepresent embodiment where the number of parallax pixels is relativelysmall, the parallax pixels have all of the combinations of the differenttypes of openings 104 and the three types of, i.e., R, G and B colorfilters. Parallax Lt pixels having an opening 104 shifted to the leftfrom the center and parallax Rt pixels having an opening 104 shifted tothe right from the center are taken as an example. The parallax Ltpixels include a pixel having an R filter, a pixel having a G filter,and a pixel having a B filter, and the parallax Rt pixels include apixel having an R filter, a pixel having a G filter, and a pixel havinga B filter. Thus, the image sensor 100 has six different types ofparallax pixels. Such an image sensor 100 outputs image data, which isused to form clear color parallax image data to realize a stereoscopicview. Note that, when two types of openings are combined with two typesof color filters, the image sensor 100 has four types of parallaxpixels.

The following describes a variation of the pixel arrangement. FIG. 7illustrates the arrangement of pixels in a repeating pattern 110relating to a first implementation. The repeating pattern 110 relatingto the first implementation includes four Bayer arrangements, each ofwhich is formed by four pixels, arranged both in the vertical direction,which is the Y-axis direction, and in the horizontal direction, which isthe X-axis direction, and is thus constituted by sixty-four pixels. Thisrepeating pattern 110 has a group of 64 pixels as a single unit, and aplurality of repeating patterns 110 are periodically arrangedhorizontally and vertically within the effective pixel region of theimage sensor 100. Thus, the repeating pattern 110 bounded by the thickbold line in FIG. 7 is the primitive lattice in the image sensor 100.Here, the pixels within the repeating pattern 110 are represented asP_(IJ). For example, the leftmost and uppermost pixel is represented asP₁₁ and the rightmost and uppermost pixel is represented as P₈₁.

Each of the parallax pixels relating to the first implementation has oneof the two types of aperture masks 103, so that the parallax pixels aredivided into the parallax Lt pixels having the openings 104 shifted tothe left from the center of the pixels and the parallax Rt pixels havingthe openings 104 shifted to the right from the center of the pixels. Asshown in FIG. 7, the parallax pixels are arranged in the followingmanner.

P₁₁ . . . parallax Lt pixel+G filter (=G(Lt))

P₅₁ . . . parallax Rt pixel+G filter (=G(Rt))

P₃₂ . . . parallax Lt pixel+B filter (=B(Lt))

P₆₃ . . . parallax Rt pixel+R filter (=R(Rt))

P₁₅ . . . parallax Rt pixel+G filter (=G(Rt))

P₅₅ . . . parallax Lt pixel+G filter (=G(Lt))

P₇₆ . . . parallax Rt pixel+B filter (=B(Rt))

P₂₇ . . . parallax Lt pixel+R filter (=R(Lt))

The other pixels are no-parallax pixels and include no-parallax pixels+Rfilter (=R(N)), no-parallax pixels+G filter (=G(N)), and no-parallaxpixels+B filter (=B(N)).

As described above, the pixel arrangement preferably includes theparallax pixels having all of the combinations of the different types ofopenings and the different types of color filters within the primitivelattice of the pixel arrangement and has the parallax pixels randomlyarranged together with the no-parallax pixels that are more than theparallax pixels. To be more specific, it is preferable, when theparallax and no-parallax pixels are counted according to each type ofcolor filters, that the no-parallax pixels are still more than theparallax pixels. In the case of the first implementation, while G(N)=28,G(Lt)+G(Rt)=2+2=4, while R(N)=14, R(Lt)+R(Rt)=2, and while B(N)=14,B(Lt)+B(Rt)=2. In addition, as described above, considering the humanspectral sensitivity characteristics, more parallax and no-parallaxpixels having the G filter are arranged than the parallax andno-parallax pixels having the other types of color filters.

Stated differently, it is attempted to obtain a higher-quality and moreaccurate color distribution structure for a stereoscopic view byacquiring information corresponding to all of the R, G and B colors alsofor the parallax pixels.

For the right parallax pixels, the left parallax pixels and theno-parallax pixels, the RGB ratio is commonly R:G:B=1:2:1, which is thesame as in the Bayer arrangement. The parallax pixels have a low densityand arranged away from each other as much as possible in order to allowthe no-parallax pixels can keep the spatial resolution at the same levelas the normal Bayer arrangement. In other words, the right parallaxpixels of a particular color component are isotropically arranged atequal intervals and the left parallax pixels of a particular colorcomponent are also isotropically arranged at equal intervals, and, atthe same time, the right parallax pixels of a particular color componentare arranged as distant as possible from the left parallax pixels of theparticular color component and the parallax pixels of the same colorcomponent are arranged at equal intervals whether they are right or leftparallax pixels. In this way, when their color components are ignored,the right parallax pixels are arranged as distant as possible from eachother and the left parallax pixels are arranged as distant as possiblefrom each other, so that parallax information can be uniformly obtained.

In the first implementation, the ratio between the number of no-parallaxpixels, the number of left parallax pixels and the number of rightparallax pixels is N:Lt:Rt=14:1:1 and the spatial resolution of theno-parallax pixels is kept at a very similar level to the spatialresolution of the Bayer arrangement. Furthermore, since the parallaxpixels are arranged as distant as possible, every parallax pixel isadjacent to a no-parallax pixel and there is no such risk that theresolutions achieved by adjacent pixels drop together. Accordingly, thefirst implementation maintains a high resolving power equivalent to highfrequency components including the Nyquist frequency components.

FIG. 8 illustrates how the pixel pitches of the various types ofparallax pixels are related to each other in the first implementation.In FIG. 8, nine repeating patterns 110, shown in FIG. 7, are arranged in3×3.

As seen from FIG. 8, the intervals between the G(Lt) pixels, which isrepresented as GLt_(p), are equal in the X direction and also equal inthe Y direction. Likewise, the intervals between the corresponding G(Rt)pixels, which is represented as GRt_(p), are equal in the X directionand also equal in the Y direction. In addition, GRt_(p) is equal toGLt_(p). Furthermore, each G(Rt) pixel is positioned at a distance ofGLt_(p) away from a G(Lt) pixel only in one of the X and Y directions.

Likewise, the intervals between the R(Lt) pixels, which is representedas RLt_(p), are equal in the X direction and also equal in the Ydirection. Likewise, the intervals between the corresponding R(Rt)pixels, which is represented as RRt_(p), are equal in the X directionand also equal in the Y direction. In addition, RRt_(p) is equal toRLt_(p). Furthermore, each R(Rt) pixel is positioned at a distance ofhalf of RLt_(p) away from a R(Lt) pixel in both of the X and Ydirections.

Furthermore, the intervals between the B(Lt) pixels, which isrepresented as BLt_(p), are equal in the X direction and also equal inthe Y direction. Likewise, the intervals between the corresponding B(Rt)pixels, which is represented as BRt_(p), are equal in the X directionand also equal in the Y direction. In addition, BRt_(p) is equal toBLt_(p). Furthermore, each B(Rt) pixel is positioned at a distance ofhalf of BLt_(p) away from a B(Lt) pixel in both of the X and Ydirections.

Thus, when the pixels are grouped according to each type of the colorfilters, the pixels having one of the types of aperture masks arearranged at equal intervals in both of the two-dimensional directionsand sandwiched between the parallax and no-parallax pixels associatedwith the other types of aperture masks. Stated differently, the pixelsassociated with each of the types of the color filters are arrangedisotropically and equally in the two-dimensional directions. Byarranging the parallax pixels in the above-described manner, parallaximages have the same resolution in both of the vertical and horizontaldirections when output and the adverse effects made by the parallaxpixels on the resolution of 2D images can be also reduced.

The above-described color-and-parallax multiplexed pixel arrangement isshown in FIG. 19A, and FIG. 19B shows a k-space to illustrate theresolving region or resolution in the frequency space corresponding tothe pixel arrangement in the real space. When k denotes the wave numberand f denotes the frequency, k=27 πf. The lattice pitch in the actualspace is denoted by a, and the reciprocal lattice space of the actualspace is represented as a k-space, and the resolving region isrepresented by a first Brillouin zone of the reciprocal lattice space(see, for example, US 2010/0201853 and Japanese Patent No. 4239483invented by the same inventor as the present application).

FIG. 19B shows that, when it comes to the images as captured, theresolving power of the left and right parallax pixels having a lowdensity is lower than the resolving power of the no-parallax pixelshaving a high density. To offset this disadvantage, the no-parallaxpixels having a high density provide for a resolving power comparable tothe resolving power of the Bayer arrangement.

Accordingly, as described later, the no-parallax pixels are firstsubject to interpolation to produce 2D color images R(N), G(N) and B(N),and low-density left parallax images R(Lt), G(Lt) and B(Lt) andlow-density right parallax images R(Rt), G(Rt) and B(Rt) are produced inadvance. The no-parallax images are used as intermediate images, so thathigh-density left parallax images R′(Lt), G′(Lt) and B′(Lt) andhigh-density right parallax images R′(Rt), G′(Rt) and B′(Rt) can befinally obtained by applying parallax modulation using the low-densityparallax images as follows.

${R^{\prime}({Lt})} = {{R(N)}\sqrt[3]{\frac{2{R({Lt})}}{{R({Lt})} + {R({Rt})}}}\sqrt[3]{\frac{2{G({Lt})}}{{G({Lt})} + {G({Rt})}}}\sqrt[3]{\frac{2{B({Lt})}}{{B({Lt})} + {B({Rt})}}}}$${G^{\prime}({Lt})} = {{G(N)}\sqrt[3]{\frac{2{R({Lt})}}{{R({Lt})} + {R({Rt})}}}\sqrt[3]{\frac{2{G({Lt})}}{{G({Lt})} + {G({Rt})}}}\sqrt[3]{\frac{2{B({Lt})}}{{B({Lt})} + {B({Rt})}}}}$${B^{\prime}({Lt})} = {{B(N)}\sqrt[3]{\frac{2{R({Lt})}}{{R({Lt})} + {R({Rt})}}}\sqrt[3]{\frac{2{G({Lt})}}{{G({Lt})} + {G({Rt})}}}\sqrt[3]{\frac{2{B({Lt})}}{{B({Lt})} + {B({Rt})}}}}$${R^{\prime}({Rt})} = {{R(N)}\sqrt[3]{\frac{2{R({Rt})}}{{R({Lt})} + {R({Rt})}}}\sqrt[3]{\frac{2{G({Rt})}}{{G({Lt})} + {G({Rt})}}}\sqrt[3]{\frac{2{B({Rt})}}{{B({Lt})} + {B({Rt})}}}}$${G^{\prime}({Rt})} = {{G(N)}\sqrt[3]{\frac{2{R({Rt})}}{{R({Lt})} + {R({Rt})}}}\sqrt[3]{\frac{2{G({Rt})}}{{G({Lt})} + {G({Rt})}}}\sqrt[3]{\frac{2{B({Rt})}}{{B({Lt})} + {B({Rt})}}}}$${B^{\prime}({Rt})} = {{B(N)}\sqrt[3]{\frac{2{R({Lt})}}{{R({Lt})} + {R({Rt})}}}\sqrt[3]{\frac{2{G({Lt})}}{{G({Lt})} + {G({Rt})}}}\sqrt[3]{\frac{2{B({Lt})}}{{B({Lt})} + {B({Rt})}}}}$$\begin{matrix}{\mspace{20mu}{Density}} & {High} & {High} & {Low} & {Low} & {Low}\end{matrix}$

In this way, the high-frequency components of the no-parallax pixels aresuperimposed to produce new parallax images, so that parallax images or3D images can achieve as high resolution as 2D images. In other words,in a slightly defocused image region in the vicinity of the focusregion, in which slight parallax is generated, the parallax modulationperforms position shifting to slightly shift in the left and rightdirections the high-resolution no-parallax images with reference to thegradual changes in the parallax images.

Furthermore, a subject image in a significantly defocused region or inthe non-focus region is significantly shifted horizontally while theresolving power of the no-parallax images is maintained as much aspossible and by making the most use of the horizontal spatial resolutionof the gradually changing parallax images.

Stated differently, the pixel arrangement is required to produceparallax images having a high horizontal spatial resolution in order tomaximize the parallax modulation effects. From this point of view, thepixel arrangement shown at the beginning in relation to the 6-viewpointexample in which the left and right parallax pixels are arranged in thehorizontal direction is not desirable due to a lowered horizontalresolution. Alternatively, a parallax pixel arrangement is required thatachieves high resolution in the horizontal direction. An isotropicparallax pixel arrangement satisfies this requirement. FIG. 19B shows ak-space representing the resolution achieved by the isotropic parallaxpixel arrangement. The following describes other exemplary pixelarrangements having parallax pixels with a low density, in which theparallax pixels are isotropically arranged. The following pixelarrangements are shown together with their k-spaces.

FIG. 9 illustrates how the pixels are arranged in a repeating pattern110 relating to a second implementation. As in the first implementation,the repeating pattern 110 relating to the second implementation includesfour Bayer arrangements, each of which is formed by four pixels, both inthe vertical direction, which is the Y-axis direction, and in thehorizontal direction, which is the X-axis direction, and is thusconstituted by sixty-four pixels. The repeating pattern 110 has a groupof 64 pixels as a single unit, and a plurality of repeating patterns 110are periodically arranged horizontally and vertically within theeffective pixel region of the image sensor 100. Thus, the repeatingpattern 110 bounded by the thick bold line in FIG. 9 is the primitivelattice in the image sensor 100.

In the second implementation, each of the parallax pixels has one of thetwo types of aperture masks 103, so that the parallax pixels are dividedinto the parallax Lt pixels having the openings 104 shifted to the leftfrom the center of the pixels and the parallax Rt pixels having theopenings 104 shifted to the right from the center of the pixels. Asshown in FIG. 9, the parallax pixels are arranged in the followingmanner.

P₁₁ . . . parallax Lt pixel+G filter (=G(Lt))

P₅₁ . . . parallax Rt pixel+G filter (=G(Rt))

P₃₂ . . . parallax Lt pixel+B filter (=B(Lt))

P₇₂ . . . parallax Rt pixel+B filter (=B(Rt))

P₂₃ . . . parallax Rt pixel+R filter (=R(Rt))

P₆₃ . . . parallax Lt pixel+R filter (=R(Lt))

P₁₅ . . . parallax Rt pixel+G filter (=G(Rt))

P₅₅ . . . parallax Lt pixel+G filter (=G(Lt))

P₃₆ . . . parallax Rt pixel+B filter (=B(Rt))

P₇₆ . . . parallax Lt pixel+B filter (=B(Lt))

P₂₇ . . . parallax Lt pixel+R filter (=R(Lt))

P₆₇ . . . parallax Rt pixel+R filter (=R(Rt))

The other pixels are no-parallax pixels and include no-parallax pixels+Rfilter (=R(N)), no-parallax pixels+G filter (=G(N)), and no-parallaxpixels+B filter (=B(N)).

As described above, the pixel arrangement preferably includes theparallax pixels having all of the combinations of the different types ofopenings and the different types of color filters within the primitivelattice of the pixel arrangement and has the parallax pixels randomlyarranged together with the no-parallax pixels that are more than theparallax pixels. To be more specific, it is preferable, when theparallax and no-parallax pixels are counted according to each type ofcolor filters, that the no-parallax pixels are still more than theparallax pixels. In the case of the second implementation, whileG(N)=28, G(Lt)+G(Rt)=2+2=4, while R(N)=12, R(Lt)+R(Rt)=4, and whileB(N)=12, B(Lt)+B(Rt)=4.

While the RGB ratio in the parallax pixel arrangement in the firstimplementation is R:G:B=1:2:1, the RGB ratio in the parallax pixelarrangement in the second implementation is R:G:B=1:1:1 by increasingthe numbers of the R and B parallax pixels to be equal to the number ofG pixels. This arrangement is realized at the sacrifice of the spatialresolution of the no-parallax pixels. FIGS. 20A and 20B show the realspace and the k space for the second implementation.

FIG. 10 illustrates how the pixels are arranged in a repeating pattern110 relating to a third implementation. As in the first and secondimplementations, the repeating pattern 110 relating to the thirdimplementation includes four Bayer arrangements, each of which is formedby four pixels, both in the vertical direction, which is the Y-axisdirection, and in the horizontal direction, which is the X-axisdirection, and is thus constituted by sixty-four pixels. The repeatingpattern 110 has a group of 64 pixels as a single unit, and a pluralityof repeating patterns 110 are periodically arranged horizontally andvertically within the effective pixel region of the image sensor 100.Thus, the repeating pattern 110 bounded by the thick bold line in FIG.10 is the primitive lattice in the image sensor 100.

In the third implementation, each of the parallax pixels has one of thetwo types of aperture masks 103, so that the parallax pixels are dividedinto the parallax Lt pixels having the openings 104 shifted to the leftfrom the center of the pixels and the parallax Rt pixels having theopenings 104 shifted to the right from the center of the pixels. Asshown in FIG. 10, the parallax pixels are arranged in the followingmanner.

P₁₁ . . . parallax Lt pixel+G filter (=G(Lt))

P₃₂ . . . parallax Lt pixel+B filter (=B(Lt))

P₆₃ . . . parallax Rt pixel+R filter (=R(Rt))

P₅₅ . . . parallax Rt pixel+G filter (=G(Rt))

P₇₆ . . . parallax Rt pixel+B filter (=B(Rt))

P₂₇ . . . parallax Lt pixel+R filter (=R(Lt))

The other pixels are no-parallax pixels and include no-parallax pixels+Rfilter (=R(N)), no-parallax pixels+G filter (=G(N)), and no-parallaxpixels+B filter (=B(N)).

As described above, the pixel arrangement preferably includes theparallax pixels having all of the combinations of the different types ofopenings and the different types of color filters within the primitivelattice of the pixel arrangement and has the parallax pixels randomlyarranged together with the no-parallax pixels that are more than theparallax pixels. To be more specific, it is preferable, when theparallax and no-parallax pixels are counted according to each type ofcolor filters, that the no-parallax pixels are still more than theparallax pixels. In the case of the third implementation, while G(N)=30,G(Lt)+G(Rt)=2, while R(N)=14, R(Lt)+R(Rt)=2, and while B(N)=14,B(Lt)+B(Rt)=2.

While the RGB ratio in the parallax pixel arrangement in the firstimplementation is R:G:B=1:2:1, the RGB ratio in the parallax pixelarrangement in the third implementation is R:G:B=1:1:1 by decreasing thenumber of G parallax pixels to be equal to the numbers of the R and Bparallax pixels. This arrangement results in enhancement of the spatialresolution of the no-parallax pixels. FIGS. 21A and 21B show the realspace and the k space for the third implementation.

In the third implementation, the parallax pixels having each type ofaperture masks are arranged so as not to overlap the other type ofaperture masks in both of the column direction (X direction) and the rowdirection (Y direction) of the two dimensional directions. To bespecific, from the perspective of the column direction, while theparallax Lt pixels are arranged in the second, third and fifth columns,the parallax Rt pixels are arranged in the first, sixth and seventhcolumns. In addition, from the perspective of the row direction, whilethe parallax Lt pixels are arranged in the second, fifth and seventhrows, and the parallax Rt pixels are arranged in the first, third andsixth rows. Thus, although the different types of the color filters arenot taken into consideration, the parallax pixels having each type ofaperture masks are arranged at equal intervals in both of the twodimensional directions. In this way, the pixels are arranged morerandomly, which enables the pixel arrangement to output high-qualityparallax images. In other words, isotropic parallax information can beobtained. This follows the arrangement rules described with reference tothe first implementation.

FIG. 11 illustrates how the pixels are arranged in a repeating pattern110 relating to a fourth implementation. The repeating pattern 110relating to the fourth implementation includes two Bayer arrangements,each of which is formed by four pixels, both in the vertical direction,which is the Y-axis direction, and in the horizontal direction, which isthe X-axis direction, and is thus constituted by sixteen pixels. Inaddition, the G filters of the Gb pixels in the upper left and lowerright Bayer arrangements are replaced with W filters that are designedto transmit the entire visible light wavelength region. The repeatingpattern 110 has a group of 16 pixels as a single unit, and a pluralityof repeating patterns 110 are periodically arranged horizontally andvertically within the effective pixel region of the image sensor 100.Thus, the repeating pattern 110 bounded by the thick bold line in FIG.11 is the primitive lattice in the image sensor 100.

In the fourth implementation, the parallax Lt pixel having the opening104 shifted to the left from the center of the pixel is at the pixelposition P₁₁ associated with the W filter, and the parallax Rt pixelhaving the opening 104 shifted to the right from the center of the pixelis at the pixel position P₃₃ associated with the W filter.

The other pixels are no-parallax pixels and include no-parallax pixels+Rfilter (=R(N)), no-parallax pixels+G filter (=G(N)), and no-parallaxpixels+B filter (=B(N)).

In the above-described arrangement, the parallax pixels having each typeof aperture masks are also arranged at equal intervals in both of thetwo dimensional directions and sandwiched between the parallax andno-parallax pixels associated with the other types of aperture masks. Inaddition, the parallax pixels having each type of aperture masks arearranged so as not to overlap the parallax pixels having the other typeof aperture masks in both of the column direction (X direction) and therow direction (Y direction) of the two dimensional directions.

FIGS. 22A and 22B show the real space and the k-space corresponding tothe above-described arrangement.

The image sensor 100 relating to the fourth implementation can produceparallax image data that provides luminance information. In other words,the image sensor 100 can output, as image data, monochrome 3D images,which can be also used as distance images used to calculate the distanceof a subject. In addition, high-resolution 2D image data is used asbasic data and multiplied by the relative ratios of the 3D image data,which is provided as luminance information, on the pixel-by-pixel basis.In this way, high-resolution color parallax image data can be produced.

The exemplary pixel arrangements described in the first to fourthimplementations follow both of the rules that parallax pixels should bearranged at a low density and isotropically. The subsequent drawings arerelated to the other possible color-and-parallax multiplexed pixelarrangements based on these arrangement rules. FIGS. 23A and 23B, FIGS.24A and 24B and FIGS. 25A and 25B respectively show the real space andthe k space corresponding to the pixel arrangements in which the colorfilters are arranged in accordance with the Bayer arrangement and theparallax pixels are assigned only to the G pixels. These pixelarrangements slightly differ from each other in terms of the density ofthe parallax pixels. FIGS. 26A and 26B, FIGS. 27A and 27B and FIGS. 28Aand 28B show the real space and the k-space for the pixel arrangementsfor monochrome sensors in which the parallax pixels are arrangedfollowing the above-described rules. Judging from the k-spacescorresponding to the respective pixel arrangements, the parallax pixelshave an isotropic frequency resolving region, the spacious resolvingregion of the no-parallax pixels is maintained, and the parallax pixelsprovide for an appropriate resolution from the perspective of thatparallax is generated in blur regions.

Likewise, FIGS. 29A and 29B, 30A and 30B, 31A and 31B and 32A and 32Brespectively show the real space and the k-space for exemplary pixelarrangements in a complementary color system. Note that C, M, Y and Wrespectively represent cyan, magenta, yellow and white.

The above describes how the parallax pixels are arranged in the primarycolor system, monochrome system, and complementary color system. Themost excellent pixel arrangement among the color pixel arrangements isthe pixel arrangement relating to the first implementation. This isbecause the pixel arrangement relating to the first implementation isbased on the Bayer arrangement and the no-parallax and parallax pixelsrealize the RGB ratio of R:G:B=1:2:1, which is equivalent to theresolution ratio approximate to the visual sensitivity characteristics,while the no-parallax pixels keep their capability at substantially thesame level as the capability of the normal Bayer arrangement.

The following describes an exemplary pixel arrangement relating to afifth implementation in which the densities of all of the R, G and Bparallax pixels are doubled compared with the pixel arrangement relatingto the first implementation. In the second implementation, only theparallax pixels of the R and B components are increased when comparedwith the pixel arrangement relating to the first implementation. In thefifth implementation, however, the parallax pixels of all of the R, Gand B components are increased so that the color distribution ratioamong the no-parallax pixels is R(N):G(N):B(N)=1:2:1, the colordistribution ratio among the left parallax pixels isR(Lt):G(Lt):B(Lt)=1:2:1, and the color distribution ratio among theright parallax pixels is R(Rt):G(Rt):B(Rt)=1:2:1, which is the samecolor distribution ratio in the Bayer arrangement, and the distributionratio between the no-parallax pixels (N), the left parallax pixels (Lt)and the right parallax pixels (Rt) is increased from N:Lt:Rt=14:1:1 toN:Lt:Rt=6:1:1.

FIG. 34A shows how the pixels are arranged in the fifth implementation.FIG. 34B shows the k-space for the pixel arrangement. However, note thatthe resolving range of the no-parallax pixels in the k-space in FIG. 34Bis assumed not to be smaller than the resolving range of the Bayerarrangement since the parallax pixels are arranged at a low density.FIG. 34B show approximately estimated resolving ranges for the G(Lt) andG(Rt) pixels.

The pixel arrangement is additionally described. In the primitivelattice of 8×8, every row has one left parallax pixel and one rightparallax pixel. Furthermore, every column has one left parallax pixeland one right parallax pixel. The parallax pixels are arranged at equalintervals and the different types of parallax pixels are arranged asdistant from each other as possible. When the left parallax pixels areconnected by straight lines irrespective of their colors, left obliquelines are drawn at approximately 30 degrees from the horizontal line andright oblique lines are also drawn orthogonally to the left obliquelines. The same applies to the right parallax pixels. Accordingly, thelow-density parallax pixels are isotropically arranged.

This pixel arrangement is characterized in that the spatial 2Dresolution and the spatial 3D resolution are well balanced. In otherwords, the no-parallax pixels are densely arranged to maintain high 2Dimage quality while the parallax pixels, which can produce stereoscopicimages, are arranged at such a density that every column and row has oneor more parallax pixels. Accordingly, the parallax pixel arrangementsrelating to the first and fifth implementations can be interpreted asbeing developed so as to be compatible with the monocular pupil-divisionstereoscopic imaging scheme while still following the color distributionratio of the Bayer arrangement.

The following describes the image processing to produce 2D image dataand a plurality of pieces of parallax image data. As seen from thearrangement of parallax and no-parallax pixels in the repeating pattern110, image data representing a particular image cannot be obtainedsimply by arranging the outputs of the image sensor 100 in accordancewith its pixel arrangement. In other words, grouping and collecting theoutputs from the pixels of the image sensor 100 for each group of pixelswith the same characteristic can provide image data representing animage having the characteristic. For example, as has been described withreference to FIG. 5, grouping and collecting the outputs from theparallax pixels according to the types of their openings can provide aplurality of pieces of parallax image data that have parallaxtherebetween. Here, the image data obtained by grouping and collectingthe outputs from the pixels for each group of the pixels having the samecharacteristic is referred to as plane data.

The image processor 205 receives raw original image data in which theoutput values of the pixels of the image sensor 100 are arranged in theorder of the pixel arrangement of the image sensor 100. The imageprocessor 205 then separates the raw original image data into aplurality of pieces of plane data. The following describes how toproduce each plane data taking, as an example, the outputs from theimage sensor 100 relating to the first implementation described withreference to FIG. 7

FIG. 12 illustrates, as an example, how to produce 2D-RGB plane data,which is 2D image data. The top drawing shows the outputs from thepixels in the single repeating pattern 110 and its surrounding pixels inthe image sensor 100 in accordance with the pixel arrangement of theimage sensor 100. Note that, in FIG. 12, the pixels are shown inaccordance with the example of FIG. 7 so that the different types ofpixels can be understood, but it is actually the output valuescorresponding to the pixels that are arranged.

To produce the 2D-RGB plane data, the image processor 205 first removesthe pixel values of the parallax pixels and creates empty pixelpositions. The pixel value for each empty pixel position is calculatedby interpolation using the pixel values of the surrounding pixels havingthe color filters of the same type. For example, the pixel value for anempty pixel position P₁₁ is calculated by averaging the pixel values ofthe obliquely adjacent G-filter pixels P⁻¹⁻¹, P²⁻¹, P⁻¹², P₂₂.Furthermore, for example, the pixel value for an empty pixel positionP₆₃ is calculated by averaging the pixel values of the R-filter pixelsP₄₃, P₆₁, P₈₃, P₆₅ that are vertically and horizontally adjacent to theempty pixel position P₆₃ with one pixel position placed therebetween.Likewise, the pixel value for an empty pixel position P₇₆ is calculatedby averaging the pixel values of the B-filter pixels P₅₆, P₇₄, P₉₆, P₇₈that are vertically and horizontally adjacent to the empty pixelposition P₇₆ with one pixel position placed therebetween.

The resulting 2D-RGB plane data obtained by the above-describedinterpolation is the same as the output from a normal image sensorhaving the Bayer arrangement and can be subsequently subjected tovarious types of processing as 2D image data. To be specific, the knownBayer interpolation technique is performed to produce color image datain which each pixel has RGB data. The image processor 205 performs imageprocessing in accordance with predetermined formats, for example,follows the JPEG standard or the like to produce still image data andfollows the MPEG standard or the like to produce moving image data.

FIG. 13 illustrates, as an example, how to produce two pieces of G planedata, which are parallax image data. In other words, GLt plane data,which is left parallax image data, and GRt plane data, which is rightparallax image data, are produced.

To produce the GLt plane data, the image processor 205 removes the pixelvalues, except for the pixel values of the G(Lt) pixels, from all of theoutput values of the image sensor 100 and creates empty pixel positions.As a result, two pixel values P₁₁ and P₅₅ are left in the repeatingpattern 110. The repeating pattern 110 is vertically and horizontallydivided into four portions. The pixel values of the 16 pixels in theupper left portion are represented by the output value at P₁₁, and thepixel values of the 16 pixels in the lower right portion are representedby the output value at P₅₅. The pixel value for the 16 pixels in theupper right portion and the pixel value for the 16 pixels in the lowerleft portion are interpolated by averaging the surrounding or verticallyand horizontally adjacent representative values. In other words, the GLtplane data has one value per 16 pixels.

Likewise, to produce the GRt plane data, the image processor 205 removesthe pixel values, except for the pixel values of the G(Rt) pixels, fromall of the output values of the image sensor 100 and creates empty pixelpositions. As a result, two pixel values P₅₁ and P₁₅ are left in therepeating pattern 110. The repeating pattern 110 is vertically andhorizontally divided into four portions. The pixel values of the 16pixels in the upper right portion are represented by the output value atP₅₁, and the pixel values of the 16 pixels in the lower left portion arerepresented by the output value at P₁₅. The pixel value for the 16pixels in the upper left portion and the pixel value for the 16 pixelsin the lower right portion are interpolated by averaging the surroundingor vertically and horizontally adjacent representative values. In otherwords, the GRt plane data has one value per 16 pixels.

In this manner, the GLt plane data and GRt plane data, which have lowerresolution than the 2D-RGB plane data, can be produced.

FIG. 14 illustrates, as an example, how to produce two pieces of B planedata, which are parallax image data. In other words, BLt plane data,which is left parallax image data, and BRt plane data, which is rightparallax image data, are produced.

To produce the BLt plane data, the image processor 205 removes the pixelvalues, except for the pixel value of the B(Lt) pixel, from all of theoutput values of the image sensor 100 and creates empty pixel positions.As a result, a pixel value P₃₂ is left in the repeating pattern 110.This pixel value is used as the representative value of the 64 pixels ofthe repeating pattern 110.

Likewise, to produce the BRt plane data, the image processor 205 removesthe pixel values, except for the pixel value of the B(Rt) pixel, fromall of the output values of the image sensor 100 and creates empty pixelpositions. As a result, a pixel value P₇₆ is left in the repeatingpattern 110. This pixel value is used as the representative value of the64 pixels of the repeating pattern 110.

In this manner, the BLt plane data and BRt plane data, which have lowerresolution than the 2D-RGB plane data, can be produced. Here, the BLtplane data and BRt plane data have lower resolution than the GLt planedata and GRt plane data.

FIG. 15 illustrates, as an example, how to produce two pieces of R planedata, which are parallax image data. In other words, RLt plane data,which is left parallax image data, and RRt plane data, which is rightparallax image data, are produced.

To produce the RLt plane data, the image processor 205 removes the pixelvalues, except for the pixel value of the R(Lt) pixel, from all of theoutput values of the image sensor 100 and creates empty pixel positions.As a result, a pixel value P₂₇ is left in the repeating pattern 110.This pixel value is used as the representative value of the 64 pixels ofthe repeating pattern 110.

Likewise, to produce the RRt plane data, the image processor 205 removesthe pixel values, except for the pixel value of the R(Rt) pixel, fromall of the output values of the image sensor 100 and creates empty pixelpositions. As a result, a pixel value P₆₃ is left in the repeatingpattern 110. This pixel value is used as the representative value of the64 pixels of the repeating pattern 110.

In this manner, the RLt plane data and RRt plane data, which have lowerresolution than the 2D-RGB plane data, can be produced. Here, the RLtplane data and RRt plane data have lower resolution than the GLt planedata and GRt plane data and substantially the same resolution as the BLtplane data and BRt plane data.

FIG. 16 is a conceptual view illustrating the relation between theresolutions of the respective planes. The 2D-RGB plane data has outputvalues whose number is substantially the same as the number of effectivepixels of the image sensor 100 since it has undergone interpolationusing, for example, the technique disclosed in US 2010-0201853. The GLtplane data and GRt plane data each have output values whose number isequal to 1/16 (¼×¼) of the number of pixels of the 2D-RGB plane data dueto the interpolation. The BLt plane data, BRt plane data, RLt plane dataand RRt plane data each have output values whose number is equal to 1/64(=⅛×⅛) of the number of pixels of the 2D-RGB plane data. These pieces oflow-resolution plane data undergo the bilinear interpolation techniqueto be enlarged with variable multiplication ratios and transformed intoplane data whose number of pixels is equal to the number of effectivepixels of the image sensor. However, these pieces of plane datasubstantially only have a resolving power equal to the resolution of theoriginal pieces of plane data before the enlargement with variablemultiplication ratios. In other words, the plane data resulting from theenlargement with variable multiplication ratios is image data in whichchanges are gradual. This fact has already been described with referenceto the k-space and merely described again from the perspective of thereal space.

Considering the differences between the resolutions of theabove-described pieces of plane data, the high-resolution 2D image canbe first output. While the information of the 2D image is used, parallaximage data is used to perform synthesis by performing parallaxmodulation using the above-described expressions. In this way, ahigh-resolution 3D image can be output.

Note that, while parallax images corresponding to the two viewpoints canbe obtained by using the two different types of parallax pixels as inthe first and second implementations, various numbers of types ofparallax pixels can be used depending on the desired number of parallaximages. Various repeating patterns 110 can be formed depending on thespecifications, purposes or the like, irrespective of whether the numberof viewpoints increases. In this case, to enable both 2D and 3D imagesto have a high resolution, it is important that the primitive lattice ofthe image sensor 100 includes parallax pixels having all of thecombinations of the different types of openings and the different typesof color filters and that the no-parallax pixels are more than theparallax pixels. Furthermore, it is also important to arrange theparallax pixels isotropically and equally.

To sum up, the nature of the present invention has the following threeimportant advantages. Firstly, when the monocular pupil-division imagingscheme is employed with parallax pixels, the parallax pixels can bearranged at a low density since parallax is only caused in a non-focusedportion or blur subject image region and the left and right parallaximages are only required to achieve a low spatial resolution. Therefore,since parallax is not caused in a focused subject image includinghigh-frequency components, no-parallax pixels can be densely arranged.Accordingly, the present invention can provide a color-and-parallaxmultiplexed pixel arrangement that is extremely suitable for themonocular pupil-division imaging scheme.

Secondly, the left and right parallax images are used to produce finalhigh-resolution color parallax images by modulating no-parallax imageshorizontally. To perform the horizontal parallax modulation mosteffectively so as to achieve a high resolution, the respective parallaximages need to have a high resolution in the horizontal direction. Thisrequirement is satisfied by the color-and-parallax multiplexed pixelarrangement in which the parallax pixels are isotropically arranged.

Thirdly, when the parallax pixels are inserted among the no-parallaxpixels, it is necessary to reduce the adverse effects made by theparallax pixels as much as possible and to maintain the spatialresolution achieved by the no-parallax pixels before the insertion ofthe parallax pixels as much as possible. This requirement is satisfiedby the method of arranging and distributing the parallax pixels asequally and isotropically as possible. For the above-described reasons,the present invention provides an image sensor of the monocularpupil-division imaging scheme in which the parallax pixels areeffectively arranged at a low density and isotropically.

In the above, the exemplary case is described in which the Bayerarrangement is employed as the color filter arrangement. It goes withoutsaying, however, other color filter arrangements can be used.Furthermore, in the above-described example, the three primary colors ofred, green and blue are used for the color filters. However, four ormore primary colors including emerald green may be used. In addition,red, green and blue can be replaced with three complementary colors ofyellow, magenta and cyan.

The above has described an example where the no-parallax pixels havefull-open masks. However, the no-parallax pixels can be also realized aspixels having half-open masks, which are the same masks as used in theparallax pixels, arranged at the center of the pixels as shown in FIG.33.

The above-described pixel arrangements in which the no-parallax pixels(N pixels) and the parallax pixels (Lt and Rt pixels) coexistadvantageously realize a wide dynamic range since the exposure thatcauses the signal amounts of the parallax pixels to be saturated isapproximately doubled when compared with the normal Bayer arrangementthat is constituted only by N pixels. Namely, the feature that theopenings to receive light are halved in the parallax pixels cansimultaneously produce the two effects that parallax is caused to enablestereoscopic imaging and that the dynamic range is increased to raisethe saturation signal amounts. Accordingly, when 2D and 3D images areproduced using the pixel arrangements described in the embodiments,high-dynamic-range images can be obtained.

In the above description, one of the parallax Lt pixel and the parallaxRt pixel is assigned to a single pixel. However, it is also possiblethat both of the parallax Lt and Rt pixels are assigned to a singlepixel. For example, the photoelectric converter element constituting asingle pixel is divided into the left and right portions, which can berespectively treated as the parallax Lt pixel and the parallax Rt pixel.In a pixel arrangement having such parallax pixels, the parallax Lt andRt pixels are arranged at a higher density and the spatial resolutionsof the parallax Lt and Rt pixels can be raised. Here, when thephotoelectric converter elements of the parallax pixels are comparedwith the photoelectric converter elements of the no-parallax pixels, oneparallax pixel has an area that is substantially half of the areaoccupied by one no-parallax pixel. Namely, square pixels, which are theN pixels, and rectangular pixels, which are the parallax Lt and Rtpixels, are mixed. When combined, one parallax Lt pixel and one parallaxRt pixels occupy a substantially square region.

FIGS. 35A and 35B show, as an example, a pixel arrangement in the realspace and the corresponding k-space. In the pixel arrangement shown inFIG. 35A, the ratio between the number of no-parallax pixels, the numberof parallax Lt pixels and the number of parallax Rt pixels isN:Lt:Rt=14:2:2. When compared with the pixel arrangement shown in FIG.19A, the densities of the parallax Lt and Rt pixels are increased. As aresult, as seen from the k-space in FIG. 35B, the spatial resolutions ofthe parallax Lt and Rt pixels are improved for all of the R, G and Bcolor components.

FIGS. 36A and 36B show, as an example, a pixel arrangement in the realspace and the corresponding k-space. In the pixel arrangement shown inFIG. 36A, the ratio between the number of no-parallax pixels, the numberof parallax Lt pixels and the number of parallax Rt pixels isN:Lt:Rt=6:2:2. When compared with the pixel arrangement shown in FIG.34A, the densities of the parallax Lt and Rt pixels are increased. As aresult, as seen from the k-space in FIG. 36B, the spatial resolutions ofthe parallax Lt and Rt pixels are improved for all of the R, G and Bcolor components.

Regarding some of the above-described pixel arrangements that satisfyspecial conditions, moving image reading can be performed by addingtogether a plurality of pixels in the horizontal direction anddiscarding a plurality of pixels in the vertical direction. FIG. 37illustrates how to perform moving image reading when the pixelarrangement shown in FIG. 34A is used. In this case, three pixels of thesame color that are adjacent to each other in the horizontal directionare added together and three pixels are discarded in the verticaldirection. In FIG. 37, for better visual understanding of the pixeladdition and discard, four unit cell arrangements, each of which is thesame as the pixel arrangement shown in FIG. 34A, are arranged in both ofthe vertical and horizontal directions. Here, the pixel position isrepresented as (i,j). For example, the leftmost and uppermost pixelposition is represented as (1,1) and the rightmost and lowermost pixelposition is represented as (32, 32).

For example, by adding together the pixel value of the GLt pixel at theposition of (1,1), the pixel value of the G pixel at the position of(1,3) and the pixel value of the GRt pixel at the position of (1,5), a Gpixel value can be obtained. Likewise, by adding together the pixelvalue of the G pixel at the position of (1,7), the pixel value of theGLt pixel at the position of (1,9) and the pixel value of the G pixel atthe position of (1,11), a GLt pixel value can be obtained. The disparityfor the GLt pixel resulting from the addition and discard is reduced to⅓ since the pixel values of one parallax pixel and two N pixels areaveraged. Therefore, the disparity may be increased three times duringthe parallax modulation. In other words, all of the modulation terms maybe multiplied three times in the case of the parallax modulation thatmaintains the differences constant and all of the modulation terms maybe raised to the third power in the case of the parallax modulation thatmaintains the ratios constant.

When the moving image reading is performed as described above and whenall pixel reading is performed, the ratio between the number ofno-parallax pixels, the number of the parallax Lt pixels and the numberof parallax Rt pixels is N:Lt:Rt=6:1:1 in both of the cases. Inaddition, the pixel arrangement in the case of the moving image readingis exactly the same as the pixel arrangement in the case of the allpixel reading if the roles of the R and B components are switched. Thus,the pixel arrangement in which the ratio between the number ofno-parallax pixels, the number of parallax Lt pixels and the number ofparallax Rt pixels is N:Lt:Rt=6:1:1 has such excellent characteristicsthat the moving image reading can be performed while the ratio ofN:Lt:Rt=6:1:1 is maintained and the relative positions of the differenttypes of pixels are not changed.

While the embodiments of the present invention have been described, thetechnical scope of the invention is not limited to the above describedembodiments. It is apparent to persons skilled in the art that variousalterations and improvements can be added to the above-describedembodiments. It is also apparent from the scope of the claims that theembodiments added with such alterations or improvements can be includedin the technical scope of the invention.

What is claimed is:
 1. An image sensor comprising: parallax pixelshaving photoelectric converter elements each of which is associated withone of a plurality of types of aperture masks that respectively haveopenings positioned to transmit different partial luminous fluxes of anincident luminous flux from each other; and no-parallax pixelsconfigured to guide the incident luminous flux to photoelectricconverter elements without limitation of openings, wherein the parallaxpixels are arranged at equal intervals in both of two dimensionaldirections in such a manner that a parallax pixel associated with anaperture mask of one of the types is sandwiched between a no-parallaxpixel and a parallax pixel associated with an aperture mask of adifferent one of the types, and parallax pixels associated with aperturemasks of different ones of the types are arranged as distant aspossible.
 2. The image sensor as set forth in claim 1, wherein each ofthe parallax and no-parallax pixels is associated with one of aplurality of types of color filters that transmit different wavelengthranges, and parallax pixels associated with each of the plurality oftypes of color filters are arranged at equal intervals in both of thetwo dimensional directions, and, among parallax pixels associated witheach of the plurality of types of color filters, parallax pixelsassociated with different ones of the plurality of types of aperturemasks are arranged as distant as possible.
 3. The image sensor as setforth in claim 1, wherein parallax pixels associated with each of theplurality of types of aperture masks are arranged so as not to overlapeach other in both of a row direction and a column direction of the twodimensional directions.
 4. An image-capturing apparatus comprising: theimage sensor as set forth in claim 1; and an image processor configuredto produce a plurality of pieces of parallax image data having parallaxtherebetween and 2D image data without parallax based on the outputsfrom the image sensor.
 5. An image sensor comprising a pixel arrangementin which: pixels constituted by photoelectric converter elements thatphotoelectrically convert incident light into electric signals areperiodically arranged on a x-y plane; at least three types of aperturemasks are provided in a one-to-one correspondence with the pixels andare respectively associated with a reference direction, a first parallaxdirection different from the reference direction and a second parallaxdirection different from the reference direction; and pixels providedwith the aperture masks associated with the first parallax direction areisotropically arranged at equal intervals in two directions of an xdirection and a y direction, each pixel provided with one of theaperture masks associated with the first parallax direction beingsandwiched between one of the pixels provided with one of the aperturemasks associated with the reference direction and one of the pixelsprovided with one of the aperture masks associated with the secondparallax direction, and pixels provided with the aperture masksassociated with the second parallax direction are isotropically arrangedat equal intervals in the two directions of the x direction and the ydirection, each pixel provided with one of the aperture masks associatedwith the first parallax direction being sandwiched between one of thepixels provided with one of the aperture masks associated with thereference direction and one of the pixels provided with one of theaperture masks associated with the first parallax direction.
 6. Theimage sensor as set forth in claim 5, wherein when the pixels areprovided with and in a one-to-one correspondence with color filters fortwo or more types of color components and at least pixels provided withcolor filters for one of the types of color components include one ormore pixels having one or more aperture masks associated with thereference direction, one or more pixels having one or more aperturemasks associated with the first parallax direction and one or morepixels having one or more aperture masks associated with the secondparallax direction, in the pixels provided with the color filters forthe one of the types of color components, the pixels having the aperturemasks associated with the first parallax direction are isotropicallyarranged at equal intervals in the two directions of the x direction andthe y direction, and the pixels having the aperture masks associatedwith the second parallax direction are isotropically arranged at equalintervals in the two directions of the x direction and the y direction.7. The image sensor as set forth in claim 6, wherein each of (i) thepixels having the aperture masks associated with the referencedirection, (ii) the pixels having the aperture masks associated with thefirst parallax direction and (iii) the pixels having the aperture masksassociated with the second parallax direction are provided for each ofthe two or more types of color components.
 8. The image sensor as setforth in claim 5, wherein in the pixels periodically arranged forming aprimitive lattice, the pixels having the aperture masks associated withthe first parallax direction are positioned in different rows and indifferent columns from each other.
 9. The image sensor as set forth inclaim 5, wherein when pixel values of all of the pixels of the pixelarrangement are read and when pixel values of the pixels in the pixelarrangement are read in such a manner that pixel values of a pluralityof pixels are added together in the x direction and a plurality ofpixels in the y direction are discarded, a density ratio between thepixels having the aperture masks associated with the referencedirection, the pixels having the aperture masks associated with thefirst parallax direction, and the pixels having the aperture masksassociated with the second parallax direction remains the same.
 10. Theimage sensor as set forth in claim 9, wherein the density ratio betweenthe pixels having the aperture masks associated with the referencedirection, the pixels having the aperture masks associated with thefirst parallax direction, and the pixels having the aperture masksassociated with the second parallax direction is 6:1:1.